Properly evaluating the ability of Video-Language Models (VLMs) to understand long videos remains a challenge. We propose a long-context video understanding benchmark, Causal2Needles, that assesses two crucial abilities insufficiently addressed by existing benchmarks: (1) extracting information from two separate locations (two needles) in a long video and understanding them jointly, and (2) modeling the world in terms of cause and effect in human behaviors. Causal2Needles evaluates these abilities using noncausal one-needle, causal one-needle, and causal two-needle questions. The most complex question type, causal two-needle questions, require extracting information from both the cause and effect events from a long video and the associated narration text. To prevent textual bias, we introduce two complementary question formats: locating the video clip containing the answer, and verbal description of a visual detail from that video clip. Our experiments reveal that models excelling on existing benchmarks struggle with causal 2-needle questions, and the model performance is negatively correlated with the distance between the two needles.

Appendix The appendix is organized as follows. We also provide details of the annotation/calibration process and the baseline neural networks (NNs) in Section D and E, respectively. We discuss results regarding each weather condition and consideration of the K-Radar dataset as a pre-training dataset for other Radar tensor datasets in Section F and G, respectively. Finally, we introduce details of devkits and list relevant URLs to help with understanding the content of the paper in Section H and I, respectively. A.1 Additional samples of the K-Radar dataset and explanation of LPCs for each weather condition In the sleet (Figure 8-(e)) or heavy snow (Figure 8-(g)) condition, the Lidar point cloud (LPC) measurements of some objects ahead are lost when the ego-vehicle is driving.

Considering different geographic regions generally have their own native sign languages, it is valuable to establish corresponding SL T datasets to support related communication and research. Auslan, as a sign language specific to Australia, still lacks a dedicated large-scale dataset for SL T.

Finally, generate a new brief description mainly based on the originalnull description with attributes information incorporated. Thenull narratives should be similar to the original description.null

Weevaluated all models onthree additional tasks, beyond those presented inthe main paper. Point-of-no-return (PNR) temporal localization error:Given a video clip of a state change, the networkhastoestimate thetimeatwhich astatechange begins. More specifically,themodel tries toestimate the keyframe within the video clip that contains the point-of-no-return (the time when the state change begins). The occurrence ofstate change isthen predicted bytraining abinary linear classifier, using the concatenated representations as input. ActionRecognition(AR)w/audio:Forthistask,videoembeddings fromfV andaudioembedding from fA are concatenated together and passed through two separate linear classifiers to classify the'verb' and'noun' of the action occurring in the video clip.

We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of over 5000 human curated multiple choice question answer pairs, spanning over 250 hours of real video data, covering a very broad range of natural human activity and behavior. For each question, EgoSchema requires the correct answer to be selected between five given options based on a three-minute-long video clip. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks & datasets.

Convolutional Neural Networks (CNNs) have been the dominant model for video action recognition. Due to the huge memory and compute demand, popular action recognition networks need to be trained with small batch sizes, which makes learning discriminative spatial-temporal representations for videos become a challenging problem. In this paper, we present Dynamic Normalization and Relay (DNR), an improved normalization design, to augment the spatial-temporal representation learning of any deep action recognition model, adapting to small batch size training settings. We observe that state-of-the-art action recognition networks usually apply the same normalization parameters to all video data, and ignore the dependencies of the estimated normalization parameters between neighboring frames (at the same layer) and between neighboring layers (with all frames of a video clip). Inspired by this, DNR introduces two dynamic normalization relay modules to explore the potentials of cross-temporal and cross-layer feature distribution dependencies for estimating accurate layer-wise normalization parameters. These two DNR modules are instantiated as a light-weight recurrent structure conditioned on the current input features, and the normalization parameters estimated from the neighboring frames based features at the same layer or from the whole video clip based features at the preceding layers. We first plug DNR into prevailing 2D CNN backbones and test its performance on public action recognition datasets including Kinetics and Something-Something. Experimental results show that DNR brings large performance improvements to the baselines, achieving over 4.4% absolute margins in top-1 accuracy without training bells and whistles.